Asynchronous transfer of messages between application programs running different data processing systems within a network is well known in the art, and is implemented by a number of commercially available messaging systems. These systems include IBM Corporation's MQSeries family of messaging products, which use asynchronous messaging via queues. A sender application program issues a PutMessage command to send (put) a message to a target queue, and MQSeries queue manager programs handle the complexities of transferring the message from the sender to the target queue, which may be remotely located across a heterogeneous computer network. The target queue is a local input queue for another application program, which retrieves (gets) the message from this input queue by issuing a GetMessage command asynchronously from the send operation. The receiver application program then performs its processing on the message, and may generate further messages. (IBM and MQSeries are trademarks of International Business Machines Corporation).
Messaging products such as MQSeries provide for assured once and once-only message delivery of messages even in the event of system or communications failures. This is achieved by not finally deleting a message from storage on a sender system until it is confirmed as safely stored by a receiver system, and by the use of sophisticated recovery facilities. Prior to commitment of transfer of the message upon confirmation of successful storage, both the deletion of the message from storage at the sender system and insertion into storage at the receiver system are kept ‘in doubt’ and can be backed out atomically in the event of a failure. This message transmission protocol and the associated transactional concepts and recovery facilities are described in international patent application WO 95/10805 and U.S. Pat. No. 5,465,328.
One key aspect of providing such an assured delivery of messages capability is the maintenance of a log in each system. A log is used to keep track of completed message activity and because a log is typically maintained a direct access non-volatile storage device (such as a hard disk drive), the information stored therein is permanently accessible. The log can therefore be used to recover the state of a system (i.e. queue(s) state) in the event of a failure thereof. Each time a recoverable message is sent to a queue, a record that the message was sent, including the message data, is written to the log, and each time a message is retrieved from a queue, a record that the message was retrieved is written to the log. In the event of a failure, the log is replayed to recover each queue to the state it was in at the point when the failure occurred.
A queue's message throughput in such a system can however be high. Further a log may be recording message activity for more than one queue. Consequently a large number of operations may be written to the log and having to replay the whole log at restart would not be feasible (this may involve replaying millions of records). In order to avoid the need to replay the entire log at restart, the queue manager periodically writes consolidated queue manager state to disk. This process is known as checkpointing. The data written at checkpoint time allows the queue manager to restart using only data that was written to the log relatively recently. A checkpoint typically hardens the current state of all queues which have changed since the previous checkpoint to permanent storage (disk).
FIGS. 1a and 1b illustrate a simplified view of logging and checkpointing according to the prior art. In FIG. 1a the state of a queue is shown in steps 1 through 8 and FIG. 1b illustrates the corresponding operations written to the log.
At step 1, messages A, B and C are put to a queue. This is recorded in the log (+A, +B, +C). An application then removes B from the queue (step 2) and this is reflected in the log (−B). Message D is subsequently added to the queue and so the queue now contains messages A, C and D (step 3). At this point the system takes a checkpoint by forcing the current state of the queue (as shown) to disk (step 4). A start checkpoint marker is placed in the log and only when checkpointing has completed is an end check point marker placed therein.
Whilst checkpointing is taking place, messages continue to be put and got from the queue and these operations are recorded in the log between the start and end markers. Thus, at step 5 message E is added to the queue and the log (+E). A is then removed from the queue (step 6) and recorded in the log (−A). At step 7, F is put to the queue and at step 8 C is removed from the queue. Once again this is all recorded in the log (+F, −C). At this point checkpointing finishes and the end checkpoint marker is placed in the log. Messages continue to be put and got from the queue and these operations recorded in the log (+G, −F). It is after F is got from the queue that the system fails and the current state of the queue (D E G) must be recovered.
As previously mentioned the whole log could be replayed in order to return the queue to its lost state. However, this would involve a large number of operations and is wasteful of both time and processing power. Thus because the consolidated state written to disk at step 4 (e.g. A C D) is recoverable, the operations stored in the log need only be replayed from the start checkpoint marker to the end of the log. Thus in this instance, A, C and D are safely on disk and only six operations have to be replayed to restore the queue. This is instead of 11 operations if the log had had to be replayed from the beginning.
Note, whilst FIG. 1a shows A, C and D being forced to disk at step 4, this is not necessarily the case. In reality what is forced to disk could be any queue state occurring between the start checkpoint marker and the end checkpoint marker. In any case checkpointing is well-known in the art and thus this should be understood by the skilled person.
Obviously FIGS. 1a and 1b show a greatly simplified view of the processing that takes place. It will be appreciated that in reality the message throughput in a messaging system will typically be far greater and that as a result the number of operations logged will be far larger. Consequently checkpoints have to be taken relatively frequently and because checkpointing involves forcing I/O it is expensive, especially considering that failures are rare occurrences and thus the majority of checkpoint data written will never be called upon.